Bugfix/s3 utils 148 mongodb improvements #300

williamlardier · 2023-10-11T14:11:50Z

bump mongodb driver by bumping arsenal version.
rework logging to display the mongodb reasons along with the error code.
add periodic logging to better track the progress of the job in case of problems.
migrate all code (including tests) to the latest mongodb changes (i.e. callbacks are removed and replaced by promises, cursors are not readable anymore by default).

bert-e · 2023-10-11T14:11:53Z

Hello williamlardier,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Status report is not available.

bert-e · 2023-10-11T14:13:50Z

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

williamlardier · 2023-10-12T12:58:12Z

/create_integration_branches

bert-e · 2023-10-12T12:58:23Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge bugfix/S3UTILS-148-mongodb-improvements into
development/1.13
w/1.14/bugfix/S3UTILS-148-mongodb-improvements will be merged into development/1.14

The following branches will NOT be impacted:

development/1.4

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

bert-e · 2023-10-12T12:58:26Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

francoisferrand · 2023-10-12T13:58:36Z

this change is incomplete : there are other modules which use mongo in this component (e.g. CrrExistingObjects, DataReport, ...) and they must be migrated as well.
this is a significant/risky change: should be done on next branch only (which is not integrated yet, I think)

williamlardier · 2023-10-12T14:11:39Z

@francoisferrand : there is no mongodb-related function in other files as far as I can see. Please share example if you saw some. Usage of already migrated functions does not require any change, as callbacks are kept; we only need to change direct use of the MongoDB driver.

We also need this change for the current branch if we want to be able to support this parameter for Artesca 1.7, if we cannot bump arsenal, then we need to create (and maintain) an hotfix branch in Arsenal, for s3utils 1.13.

francoisferrand · 2023-10-13T12:59:01Z

Indeed, I missed the fact we did not use mongo driver directly, but through arsenal which has been migrated!

We also need this change for the current branch if we want to be able to support this parameter for Artesca 1.7, if we cannot bump arsenal, then we need to create (and maintain) an hotfix branch in Arsenal, for s3utils 1.13.

It would be much better to merge on 1.14, and switch to s3utils 1.14 on Artesca : this way we keep the option to stick or revert to 1.13, and do not mix concerns.

As far as "product" is concerned, it should not matter that we make the change on 1.13 or 1.14 : they will get the same code either way.... but as far as "engineering" (and maintenance) is concerned, this is the safe way.

benzekrimaha

LGTM

francoisferrand · 2023-10-24T15:27:52Z

utils/S3UtilsMongoClient.js

                res => {
+                    // Periodically display information about the cursor
+                    // if more than 30s elapsed
+                    if (Date.now() - startCursorDate > 30000) {


this will call each time the callback is called, once the 30 second liimit has been reached: need to reset startCursorDate when we add a log ?

Suggested change

if (Date.now() - startCursorDate > 30000) {

const currentDate = Date.now();

if (currentDate - startCursorDate >= 30000) {

startCursorDate = currentDate;

Done in 8f7df21

I rebased and squashed the commits, see the updated link: https://github.com/scality/s3utils/pull/300/files#diff-3a778b361ddfba6106347ff0cb053bcbb1a84374fa0d7e6e6413e7c6fb6ca5cdR55

utils/S3UtilsMongoClient.js

williamlardier · 2023-10-27T09:35:35Z

/approve

bert-e · 2023-10-27T09:35:44Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/1.14

The following branches will NOT be impacted:

development/1.13
development/1.4

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve, create_integration_branches

bert-e · 2023-10-27T09:35:53Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/1.14

The following branches have NOT changed:

development/1.13
development/1.4

Please check the status of the associated issue S3UTILS-148.

Goodbye williamlardier.

scality deleted a comment from bert-e Oct 11, 2023

williamlardier force-pushed the bugfix/S3UTILS-148-mongodb-improvements branch from d7a99a3 to 4c1926a Compare October 12, 2023 07:09

williamlardier marked this pull request as ready for review October 12, 2023 12:58

williamlardier mentioned this pull request Oct 12, 2023

Bugfix/zenko 4668 fix arsenal memory leak scality/Zenko#1968

Merged

williamlardier requested review from anurag4DSB and benzekrimaha October 12, 2023 13:09

williamlardier requested a review from francoisferrand October 12, 2023 14:33

williamlardier changed the base branch from development/1.13 to development/1.14 October 13, 2023 13:39

scality deleted a comment from bert-e Oct 16, 2023

benzekrimaha approved these changes Oct 20, 2023

View reviewed changes

KillianG approved these changes Oct 24, 2023

View reviewed changes

francoisferrand reviewed Oct 24, 2023

View reviewed changes

utils/S3UtilsMongoClient.js Show resolved Hide resolved

williamlardier force-pushed the bugfix/S3UTILS-148-mongodb-improvements branch from 862a619 to 8f7df21 Compare October 25, 2023 10:02

williamlardier requested a review from francoisferrand October 26, 2023 07:55

williamlardier added 5 commits October 26, 2023 10:46

S3UTILS-148: bump arsenal version

e8fbce9

S3UTILS-148: update mongodb callbacks to promises

b382a4f

S3UTILS-148: improve the count items logging

9ac7df6

S3UTILS-148: fix test framework issue

32400d0

S3UTILS-148: bump project version

abe7c14

williamlardier force-pushed the bugfix/S3UTILS-148-mongodb-improvements branch from 8f7df21 to abe7c14 Compare October 26, 2023 08:47

scality deleted a comment from bert-e Oct 26, 2023

bert-e merged commit abe7c14 into development/1.14 Oct 27, 2023
12 checks passed

bert-e deleted the bugfix/S3UTILS-148-mongodb-improvements branch October 27, 2023 09:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bugfix/s3 utils 148 mongodb improvements #300

Bugfix/s3 utils 148 mongodb improvements #300

williamlardier commented Oct 11, 2023 •

edited

Loading

bert-e commented Oct 11, 2023

bert-e commented Oct 11, 2023

williamlardier commented Oct 12, 2023

bert-e commented Oct 12, 2023

bert-e commented Oct 12, 2023

francoisferrand commented Oct 12, 2023

williamlardier commented Oct 12, 2023 •

edited

Loading

francoisferrand commented Oct 13, 2023

benzekrimaha left a comment

francoisferrand Oct 24, 2023 •

edited

Loading

williamlardier Oct 25, 2023

williamlardier Oct 26, 2023

williamlardier commented Oct 27, 2023

bert-e commented Oct 27, 2023

bert-e commented Oct 27, 2023

-                    if (Date.now() - startCursorDate > 30000) {
+                    const currentDate = Date.now();
+                    if (currentDate - startCursorDate >= 30000) {
+                        startCursorDate = currentDate;

Bugfix/s3 utils 148 mongodb improvements #300

Bugfix/s3 utils 148 mongodb improvements #300

Conversation

williamlardier commented Oct 11, 2023 • edited Loading

bert-e commented Oct 11, 2023

Hello williamlardier,

bert-e commented Oct 11, 2023

Request integration branches

williamlardier commented Oct 12, 2023

bert-e commented Oct 12, 2023

Integration data created

bert-e commented Oct 12, 2023

Waiting for approval

francoisferrand commented Oct 12, 2023

williamlardier commented Oct 12, 2023 • edited Loading

francoisferrand commented Oct 13, 2023

benzekrimaha left a comment

Choose a reason for hiding this comment

francoisferrand Oct 24, 2023 • edited Loading

Choose a reason for hiding this comment

williamlardier Oct 25, 2023

Choose a reason for hiding this comment

williamlardier Oct 26, 2023

Choose a reason for hiding this comment

williamlardier commented Oct 27, 2023

bert-e commented Oct 27, 2023

In the queue

bert-e commented Oct 27, 2023

williamlardier commented Oct 11, 2023 •

edited

Loading

williamlardier commented Oct 12, 2023 •

edited

Loading

francoisferrand Oct 24, 2023 •

edited

Loading